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MEMORY EFFICffiNT METHOD AND APPARATUS FOR DISPLAYING 
LARGE OVERLAID CAMERA IMAGES 

This invention relates to a memory efficient method and apparatus for displaying 
5 large overlaid camera images, particularly for use in an electronic computing device such 
as a cellular telephone. 

Background 

It is common in electronic computing systems or devices having graphics displays, 
1 0 such as cell phones, to overlay one image on top of another. Typically, the main, or 
underlying image is covered by the overlay image so that, where the main and overlay 
images overlap, the main image is not visible. One particular example of overlaying 
occurs where it is'clesired to provide a picture frame or border around an image. 
Similarly, other geometric shapes or text may be superimposed over an image. 
15 In such applications, the overlay image defines pixels that are transparent and 

pixels that are not transparent, or opaque. Where the overlay image is transparent, the 
underlying main image is visible, and where the overlay image is opaque, only the 
overlay image is visible. Pixels in the overlay image can be defined as being transparent 
or opaque by the use of appropriate coding. For example, particular colors can be 
20 defined as being transparent while other colors can be defined as being opaque. 

In many cell phones, the main image is produced by a camera. A graphics 
controller includes a camera interface for interfacing to the camera. The graphics 
controller resizes the camera data and converts the camera data from YUV to RGB color 
fomiat for storage in an on-board display buffer. The graphics controller also receives an 
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overlay image from a host CPU and stores the overlay image data in the display bufTer as 
well. The overlay image and the main image may or may not be the same size. 

The graphics controller fetches a main image pixel and a corresponding overlay 
image pixel from the display buffer and combines the two pixels. This sequence is 
5 performed for every pixel in the main image, to form composite image data. The 
composite image data are provided to an interface for interfacing to a display device. 

One problem with the methodology described above is that it requires a 
significant amount of memory, because both images are stored in the display buffer. This 
problem is exacerbated if the overlay image is the same size as the main image, so that 
10 twice the amoimt of memory required for the main image is required for the composite. 
Another problem with the methodology is that a main image which is too large to fit 
within the display buffer cannot be displayed. 

Accordingly, there is a need for a memory efficient method and apparatus for 
displaying large overlaid camera images that solves these problems. 

15 

Summary 

A memory efficient method and apparatus for displaying large overlaid camera 
images according to the invention combines main image data and overlay image data to 
fomi composite image data for rendering on a graphics display device. According to one 
20 aspect of the invention, the overlay image data are stored in a memory, fetched, up-scaled, 
and then combined with the main image data to form the composite image data. 
According to another aspect of the invention, the overlay image data are stored in a 
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memory, fetched, and then combined with main image data streamed from a source of the 
main image data. 

Brief Description of the Drawings 
5 Figure 1 is block diagram of a prior art electronic computing system. 

Figure 2 is a block diagram of an electronic computing system for displaying 
overlaid camera images according to the present invention. 

Figure 3 is a block diagram of an altemative electronic computing system for 
displaying large overlaid camera images according to the present invention. 
10 Figure 4 is a schematic representation of up-scaling down-scaled overlay image 

data according to the present invention. 

Detailed Description of Preferred Embodiments 

Figure 1 shows a prior art electronic computing system 10 for comparison with 

15 systems described herein according to the present invention. The system 10 includes a 
graphics controller 12 that interfaces to (a) a camera 14, (b) a host CPU 16, and (c) a 
graphics display device 18 such as an LCD panel. The graphics controller manages 
image data streaming from the camera and the CPU so as to combine the image data into 
a composite, and provides the composite data to the graphics display device. 

20 The composite image is particularly a main image overlaid with an overlay image. 

Main image data defining the main image is provided by the camera 14, and overlay 
image data defining the overlay image is provided by the CPU 16. The overlay image 
data function as a template through which the main image data are to be viewed. 
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Main image data obtained from the camera are received by the camera interface 
(IF) 20. The received data are provided to a resizer 22 and, in turn, a color format 
converter 24 for converting the (typically) YUV camera output to RGB format for 
storage in a first memory portion "Mi "of a display buffer 26. Other color conversions 
5 may also be performed, and these functions can be performed in altemative sequences. 

The graphics controller 12 stores overlay image data received from the host CPU 
16 directly to a second memory portion "M2" of the display buffer 26, separate from the 
first portion. The graphics controller includes a fetching module 23 for fetching, pixel by 
pixel, each of the main image data and the corresponding overlay image data from the 
10 display buffer 26. The fetched data are provided to a graphics display interface 28. The 
graphics display interface includes a combining circuit 29 for forming a composite image 
from the main and overlay image data. The graphics display interface transmits the 
composite image to the graphics display device for display. 

The interface 28 receives, for each pixel location on the display, a main image 
15 pixel and an overlay image pixel. Where the overlay image pixel is transparent, the 
combining circuit 29 selects the main image pixel for the composite image. Where the 
overlay image pixel is opaque, the combining circuit selects the overlay image pixel for 
the composite image. 

Turning to Figure 2, an electronic computing system 30 for displaying overlaid 
20 camera images according to the present invention is shown. Like the system 10, the 

system 30 includes a graphics controller 32 that interfaces to a camera 34, a host CPU 36, 
and a graphics display device 38. In the context of the present invention, a graphics 
display device may be any device adapted for rendering image data. A preferred graphics 
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display device is an LCD panel, but the graphics display device may be, for example, a 
CRT, plasma display, OLED, printer, or equivalent device. 

As in the graphics controller 12 of the system 10, an objective of the system 30 is 
to form a composite image from a main image and an overlay image. In the preferred 
5 cell phone context of the invention, the overlay image data are generated or stored in the 
CPU 36 while the main image data are generated by the camera 34; however, any source 
of graphics output may be used to produce either type of image data without departing 
from the principles of the invention. 

Also as in the graphics controller 12, main image data obtained from the camera 

10 are received by a camera interface (IF) 40. The received data are streamed through a 
transmitting pipe 41 to a graphics display interface 48 for interfacing to the graphics 
display device 38. The transmitting pipe 41 typically includes processing units such as a 
resizer 42 and a color format converter 44 as in the controller 12; however, such 
processing units are not essential to the invention. 

15 The graphics controller 32 stores overlay image data received from the host CPU 

36 in an internal display buffer 46. However, in contradistinction to the graphics 
controller 12 and in accordance with principles of the present invention, the main image 
data are transmitted by the transmitting pipe 41 from the camera interface 40 to the 
graphics display interface 48 without fetching the data as in the prior art. Such 

20 transmitting is referred to herein as "streaming." For example, the aforementioned 

processing units receive ordered pixel data, so that there is no need to address a memory 
to obtain the pixel data. Accordingly, to perform streaming according to the present 
invention, the transmitting pipe 41 does not require a fetching module. 
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The graphics controller includes a fetching module 43 for fetching overlay image 
pixels from the display buffer 46 and provides the overlay image pixels to the interface 
48 as corresponding main image pixels are streamed from the color converter 44 to the 
interface 48. A composite image is formed at the interface using a combining circuit 49 
5 that is the same as or similar to that described above in connection with the graphics 
controller 12. However, because the main image data are not stored, e.g., in the display 
buffer 46, the display buffer can be smaller, providing a commensurate reduction in 
power consumption and increased speed of operation. 

The main image data correspond to main image pixels and the overlay image data 
10 correspond to overlay image pixels. The display interface 48 includes a counter for 
counting the main image pixels corresponding to main image data received from the 
transmitting pipe 41 and conmiunicates with the fetching module 43 so as to fetch the 
corresponding overlay image pixel from the overlay image data stored in the display 
buffer 46. Synchronization of the main and overlay image pixels may be accomplished 
1 5 by any means known in the art as will be readily ^predated by persons of ordinary skill. 

The gr^hics display interface 48 typically transmits the composite image to the 
graphics display device for display; however, this not essential. 

Streaming the main image data within the gr^hics controller 32 from the camera 
interface 40 to the graphics display interface 48 for combining with the overlay image for 
20 provision of the composite data to the graphics display device is generally practical only 
if the data meet the timing requirements of the graphics display device. That is, a timing 
adaptation would typically be required if the graphics display device does not have its 
own memory. However, in the preferred cell phone context of the invention as well as in 
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other contexts, LCD panels are now being provided with internal RAM (SSa). 
Accordingly, the main image data can be provided to the panels, where the data are 
written to the internal RAM, so that the panels can read the data from the internal RAM 
with the appropriate timing. 
5 Turning now to Figure 3, an enhanced electronic computing system 50 according 

to the present invention that is particularly advantageous for displaying large overlaid 
camera images is shown. Like the system 30, the system 50 includes a graphics 
controller 52 that interfaces to a camera 54, a host CPU 56, and a graphics display device 
58. 

10 As for the system 30, in the preferred cell phone context of the invention, the 

overlay image data are generated or stored in the CPU 56 while the main image data are 
generated by the camera 54; however, any source of graphics output may be used to 
produce either type of image data without departing from the principles of the invention. 
As in the graphics controller 32, main image data obtained from the camera are 

15 received by the camera interface (IF) 60 and streamed through a transmitting pipe 61 to a 
graphics display interface 68 provided for interfacing to the graphics display device 58. 
The transmitting pipe 61 typically includes processing units such as a resizer 62 and a 
color format converter 64, however, such processing xmits are not essential to the 
invention. 

20 Also as in the graphics controller 32, overlay image data received from the host 

CPU 56 are stored in the graphics controller 52 in an internal display buffer 56. However, 
according to principles of the invention, it is recognized that the resolution of the overlay 
image data is not critical, since they represent artificial features, such as borders or text. 
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that typically do not have or require a high resolution. In accord with this recognition, 
the overlay image according to the invention is provided as a scaled down version of the 
desired overlay image as displayed on the display device. This permits a further 
reduction in the size of the display buffer and the performance consequences thereof. 
5 Accordingly, a fetching module 63 fetches the scaled down overlay image data 

from the display buffer 66 and the fetched data are provided to an scaling circuit 67 that 
up-scales the overlay image data before providing the overlay image data to the graphics 
display interface 68. The amount that the overlay image data are down-scaled for storage 
is typically known in advance and the scaling circuit 67, in up-scaling the overlay image 

10 data, restores the overlay image data to full size with reference to a suitable scale factor. 
Subsequently, as in the graphics controller 32, a composite (main + overlay) image is 
formed at the interface using a combining circuit 69 that is the same as or similar to that 
described above in connection with the graphics controller 12. The graphics display 
interface 68 typically transmits the composite image to the graphics display device for 

15 display; however, this not essential. 

Figure 4 shows an example of a border defining, for illustrative purposes only, a 
16 pixel scaled down overlay image of a border comprising 12 border pixels that are 
opaque ("O") and 4 interior pixels that are transparent ("T"). The 16 pixel scaled down 
overlay image scales up in this example to a 64 pixel image. Particularly, a single comer 

20 pixel C( 1,1) 16 is expanded by up-scaling to become the group of 4 pixels indicated as 
C(l-2, 1-2)64 and a single interior pixel "T(2,2)i6" is expanded to become the group of 
pixels indicated as C(3-4, 3-4)64. In this example and as is typical, it can be readily 
appreciated that no loss of resolution occvirs as a result of the scaling. It will also be 
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appreciated that scaling may be by any known method, e.g., any integral or fractional 
zoom. 

As noted above, the system 50 is particularly advantageous where the overlay 
image is large, and may even be necessary where the overlay image is larger than the 
main image. However, the system 50 provides a similar advantage for displaying overlay 
images that are the same size as, or that are smaller than, the corresponding main image. 
Additional processing may be performed on the composite image data without departing 
from the principles of the invention. 

It should be recognized that, while specific memory efficient methods and 
apparatus for displaying large overlaid camera images have been shown and described as 
preferred, other configurations and methods could be utilized, in addition to those already 
mentioned, without departing from the principles of the invention. For example, 
wherever it is desired, overlay image data may be substituted for main image data and the 
reverse. 

The terais and expressions which have been employed in the foregoing 
specification are used therein as terms of description and not of limitation, and there is no 
intention in the use of such terms and expressions to exclude equivalents of the features 
shown and described or portions th^eof, it being recognized that the scope of the 
invention is defined and limited only by the claims that follow. 
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